This article digs into why repeated LLM calls can produce different outputs even at zero temperature. It shows that floating-point non-associativity and kernel implementation details—rather than thread scheduling or atomic adds—are the real sources of run-to-run variation and outlines ways to make inference fully reproducible.

• Bit-identical results across thousands of runs of the same GPU matmul disprove the standard "concurrency + floating-point non-associativity" explanation for LLM nondeterminism.
• The actual cause is that libraries like cuBLAS/cuDNN nondeterministically pick among multiple reduction/tiling algorithms at runtime, each summing floats in a different order.
• Forcing a single deterministic algorithm choice (via flags like CUBLAS_WORKSPACE_CONFIG, torch.use_deterministic_algorithms(True), and seeded CUDA RNG) makes LLM inference output bit-for-bit reproducible.